Method and apparatus for interactive reports

ABSTRACT

Methods, apparatus and computer program products for interactive reports are disclosed herein. In some examples, a method for generating an output report that is an interactive report may include identifying one or more messages to be hyperlinked in an output report, wherein the one or more messages are data structures that are configured to linguistically describe at least a portion of raw input data; determining one or more interactive responses based on the one or more messages to be hyperlinked; determining one or more words in a phrase specification that are related to the one or more messages to be hyperlinked; and generating the output report, wherein the one or more words are hyperlinked in the output report such that when selected at least one of the one or more interactive responses is performed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/961,222, titled “METHOD AND APPARATUS FOR INTERACTIVE REPORTS,” filedDec. 7, 2015, which is a continuation of U.S. application Ser. No.14/027,775, titled “METHOD AND APPARATUS FOR INTERACTIVE REPORTS,” filedSep. 16, 2013, now U.S. Pat. No. 9,244,894, the contents of which areincorporated herein by reference in their entirety.

TECHNOLOGICAL FIELD

Embodiments of the present invention relate generally to naturallanguage generation technologies and, more particularly, relate to amethod, apparatus, and computer program product for providinginteractive reports.

BACKGROUND

Natural language generation (NLG) is sometimes referred to as a subfieldof artificial intelligence and computational linguistics that focuses onthe production of understandable texts in English or otherunderstandable language. In some examples, a natural language generation(NLG) system is configured to transform raw input data that is expressedin a non-linguistic format into a format that can be expressedlinguistically, such as through the use of natural language (e.g., theconversion from data to text). In some cases the data is high frequencynumerical data. For example, raw input data may take the form of a valueof a stock market index over time and, as such, the raw input data mayinclude data that is suggestive of a time, a duration, a value and/orthe like. Other examples, may include the generation of textual weatherforecasts base on numerical weather prediction data. Therefore, an NLGsystem may be configured to input the raw input data and output textthat linguistically describes the value of the stock market index; forexample, “securities markets rose steadily through most of the morning,before sliding downhill late in the day.” Importantly, for use in an NLGsystem, data must be analysed and interpreted in a way in which theanalysis and interpretation can be linguistically communicated. Forexample, data that indicates the price of a stock market rising may berepresent linguistically as rising, spiking or the like. A human maythen make decisions based on how that human interprets rising versusspiking.

Data that is input into a NLG system may be provided in, for example, arecurrent formal structure. The recurrent formal structure may comprisea plurality of individual fields and defined relationships between theplurality of individual fields. For example, the input data may becontained in a spreadsheet or database, presented in a tabulated logmessage or other defined structure, encoded in a ‘knowledgerepresentation’ such as the resource description framework (RDF) triplesthat make up the Semantic Web and/or the like. In some examples, thedata may include numerical content, symbolic content or the like.Symbolic content may include, but is not limited to, alphanumeric andother non-numeric character sequences in any character encoding, used torepresent arbitrary elements of information. In some examples, theoutput of the NLG system is text in a natural language (e.g. English,Japanese or Swahili), but may also be in the form of synthesized speech.

BRIEF SUMMARY

In some example embodiments, a computer implemented method is disclosedherein that includes identifying one or more messages to be hyperlinkedin an output report, wherein the one or more messages are datastructures that are configured to linguistically describe at least aportion of raw input data; determining one or more interactive responsesbased on the one or more messages to be hyperlinked; determining one ormore words in a phrase specification that are related to the one or moremessages to be hyperlinked; and generating the output report, whereinthe one or more words are hyperlinked in the output report such thatwhen selected at least one of the one or more interactive responses isperformed.

In some example embodiments, an apparatus is disclosed herein thatincludes at least one processor; and at least one memory includingcomputer program code, the at least one memory and the computer programcode configured to, with the at least one processor, cause the apparatusto at least identify one or more messages to be hyperlinked in an outputreport, wherein the one or more messages are data structures that areconfigured to linguistically describe at least a portion of raw inputdata; determine one or more interactive responses based on the one ormore messages to be hyperlinked; determine one or more words in a phrasespecification that are related to the one or more messages to behyperlinked; and generate the output report, wherein the one or morewords are hyperlinked in the output report such that when selected atleast one of the one or more interactive responses is performed.

In some example embodiments, a computer program product is disclosedherein that includes at least one computer readable non-transitorymemory medium having program code instructions stored thereon, theprogram code instructions which when executed by an apparatus cause theapparatus at least to identify one or more messages to be hyperlinked inan output report, wherein the one or more messages are data structuresthat are configured to linguistically describe at least a portion of rawinput data; determine one or more interactive responses based on the oneor more messages to be hyperlinked; determine one or more words in aphrase specification that are related to the one or more messages to behyperlinked; and generate the output report, wherein the one or morewords are hyperlinked in the output report such that when selected atleast one of the one or more interactive responses is performed.

In some example embodiments, an apparatus is disclosed herein thatincludes means for identifying one or more messages to be hyperlinked inan output report, wherein the one or more messages are data structuresthat are configured to linguistically describe at least a portion of rawinput data; means for determining one or more interactive responsesbased on the one or more messages to be hyperlinked; means fordetermining one or more words in a phrase specification that are relatedto the one or more messages to be hyperlinked; and means for generatingthe output report, wherein the one or more words are hyperlinked in theoutput report such that when selected at least one of the one or moreinteractive responses is performed.

In some example embodiments, a computer implemented method is disclosedherein that includes displaying an output report having one or morehyperlinks surrounding one or more words, wherein the one or morehyperlinks provide an indication that an interactive response isavailable; receiving an indication of a selection of a hyperlink of theone or more hyperlinks; determining a communicative goal for a sentencehaving the hyperlink and a current context of the reader; and displayingan interactive response.

In some example embodiments, an apparatus is disclosed herein thatincludes at least one processor; and at least one memory includingcomputer program code, the at least one memory and the computer programcode configured to, with the at least one processor, cause the apparatusto at least display an output report having one or more hyperlinkssurrounding one or more words, wherein the one or more hyperlinksprovide an indication that an interactive response is available; receivean indication of a selection of a hyperlink of the one or morehyperlinks; determine a communicative goal for a sentence having thehyperlink and a current context of the reader; and display aninteractive response.

In some example embodiments, a computer program product is disclosedherein that includes at least one computer readable non-transitorymemory medium having program code instructions stored thereon, theprogram code instructions which when executed by an apparatus cause theapparatus at least to display an output report having one or morehyperlinks surrounding one or more words, wherein the one or morehyperlinks provide an indication that an interactive response isavailable; receive an indication of a selection of a hyperlink of theone or more hyperlinks; determine a communicative goal for a sentencehaving the hyperlink and a current context of the reader; and display aninteractive response.

In some example embodiments, an apparatus is disclosed herein thatincludes means for displaying an output report having one or morehyperlinks surrounding one or more words, wherein the one or morehyperlinks provide an indication that an interactive response isavailable; means for receiving an indication of a selection of ahyperlink of the one or more hyperlinks; means for determining acommunicative goal for a sentence having the hyperlink and a currentcontext of the reader; and means for displaying an interactive response.

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described embodiments of the invention in general terms,reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and wherein:

FIG. 1 is a schematic representation of an interactive report generationenvironment that may benefit from some example embodiments of thepresent invention;

FIG. 2 illustrates an example document plan tree and a textspecification in accordance with some example embodiments of the presentinvention;

FIG. 3 illustrates a block diagram of an apparatus that embodies aninteractive report generation environment in accordance with someexample embodiments of the present invention; and

FIGS. 4-7 illustrate flowcharts that may be performed by an interactivereport generation environment in accordance with some exampleembodiments of the present invention.

DETAILED DESCRIPTION

Example embodiments will now be described more fully hereinafter withreference to the accompanying drawings, in which some, but not allembodiments are shown. Indeed, the embodiments may take many differentforms and should not be construed as limited to the embodiments setforth herein; rather, these embodiments are provided so that thisdisclosure will satisfy applicable legal requirements. Like referencenumerals refer to like elements throughout. The terms “data,” “content,”“information,” and similar terms may be used interchangeably, accordingto some example embodiments, to refer to data capable of beingtransmitted, received, operated on, and/or stored. Moreover, the term“exemplary”, as may be used herein, is not provided to convey anyqualitative assessment, but instead merely to convey an illustration ofan example. Thus, use of any such terms should not be taken to limit thespirit and scope of embodiments of the present invention.

Overview

In NLG implementations, a distinction has been drawn between dialogicsystems and monologic systems. A dialogic system may take the form of adialog system, such as a speech telephony based system, where a userinteracts with a machine either via text or voice and then that user mayreceive a text or voice response from the machine. The dialogue maycontinue, in some examples, until a particular result is reached oruntil the user or machine ends the interaction. Examples of monologicsystems generally include NLG systems that may be configured to generatea report or generate some other form of text without interaction withthe user or with limited interaction with the user (e.g., a request fora report to be generated). A traditional monologic system can be viewedas a short dialogue that comprises a user requesting a report and thesystem responding by generating the report. In a monologic NLG systemthat generates a textual report based on a dataset, there is generally asingle question (e.g., generate a report) and a single answer (e.g., thereport). In such systems, a user may be left with follow up questions tothe report or may desire additional awareness of the situation (e.g.,additional elaboration or context) described in the report.

As such, the methods and apparatus described herein are configured togenerate interactive reports by generating NLG reports (e.g., answers)or interactive reports in such a way that enables a user to interactfurther with the NLG system. In some examples, an interactive report maybe configured such that a user may click on a sentence in the reportand, in response, have a graph generated that elaborates on the contentof the sentence. For example, if a sentence that talks about how thetemperature rose over some period of time is selected (e.g., thequestion), then a graph associated with the temperature rise may bedisplayed (e.g., the answer). Advantageously, by enabling the selectionof the sentence and the subsequent generation of the graph, the methodsand apparatus described herein have added a further conversational turn(e.g., a question and answer pair) to what otherwise may be a monologicNLG system

In order to generate the interactive or subsequent response (e.g., theadditional answer in the form of any one or more graphs, reports,speech, visualizations or the like) the method and apparatus areconfigured to generate or otherwise determine the communicative intentor goal of a particular portion of the output report. For example, areport may describe one or more data streams related to a compressor. Inthe example report, a sentence may be included that describestemperature readings over a period of time. Such a sentence might bedisplayed in the form of a hyperlink that functions like a request forfurther information in a dialog by generating a more detailed reportabout the temperature upon selection of the hyperlink or alternativelythe sentence may be selectable through a graphical user interface, auser may interact with the sentence by selecting the type of responsethe user wants to receive (e.g., more detailed report, a reportjustifying the position taken in the sentence, voice output, pictorialrepresentation, graph or the like) or the like. A selection of thesentence is, in some examples, a request to the NLG system to expand orotherwise elaborate on the content of the particular sentence. As such,the subsequent report may contain more elaborative responses, such asmore detail on the temperature, an explanation of the importance of thetemperature, a historical analysis of the temperature or the like.

In some examples, the methods and apparatus described herein may alsotake into account a context of the reader when generating subsequentresponses (e.g., interactive response, another report or the like), suchas what the reader has already read in the current report, historicalreports, the last time the reader read about the particular piece ofequipment described in the report or the like. As such, the method andapparatus described herein may generate subsequent responses based onthe context of the portion of the text selected within the report,previously generated reports or the like. In other examples, context mayinclude the context in which the reader analyzes information.

For example, a user may select a sentence related to a temperature of acompressor. In response to that interaction, the system may display atemperature graph but may also show the weather on that particular day,the temperature of a related compressor or the like in the same graph toprovide the reader with context as to why the temperature sentence wasimportant, to explain the temperature sentence, to provide the user withincreased situational awareness or the like.

Context may also include not telling or not repeating in a subsequentresponse something that the reader has already been told. For example,if a user was previously told about or otherwise shown a relatedtemperature in response to a previous click then that relatedtemperature information may be omitted in the response to a subsequentclick. In response to the subsequent click additional or alternateinformation may be described. In some examples, the additionalinformation may be generated from messages not included in an originaldocument plan.

In another example, a graphical output may be dynamically generatedbased on the context (e.g., based on data captured or otherwise definedby the one or more messages that were used to generate the portion oftext selected). For example, if a report describes a stock portfolio bytalking about how the portfolio performed and then it talks aboutindividual stocks, if a user clicks on the text in the beginning of thereport that talks about portfolios, a graph may be displayed that has aline or other indication of a performance for each stock. Whereas, ifthe same user did not click on the description of the portfolio butinstead just selected the particular stock later in the report, thatparticular stock may be displayed with the remaining stocks in theportfolio or a graphical representation of the portfolio performancedisplayed in the background. As such, the resulting graph generated maybe different depending on what graphs have already been shown to theuser.

In some examples, portions of the output text may be highlighted,underlined or otherwise indicative (e.g., by using meta-tags) of anability to be selected to enable elaboration. For example, a sentencemay state that the temperature rose steeply between 5 am and 6 am. Inone example, the entire sentence might be a hyperlink and clicking onthe hyperlink that is tied to that text is considered to be a requestfor the NLG system to produce a subsequent report to explain temperaturerise. However, in another example, only the terms ‘5 am and 6 am’ couldbe hyperlinked such that a graph may be displayed showing temperaturesat a given time or alternatively may provide context to explain anyrelated details of the temperature at the selected time. In other cases,the selection of a word or phrase may provide for the definition of theword or phrase in the context that the word or phrase is used in thereport. In further examples, the selection of sentence might beequivalent to a request to explain the sentence or justify the claimmade in the sentence. Each type of response provided by the NLG systemmay be defined by the system or may, in some examples, be defined by theuser.

Natural Language Generation System

FIG. 1 is an example block diagram of example components of an exampleinteractive report generation environment 100 that may take the form of,for example, a code module, a component, circuitry and/or the like. Thecomponents of the interactive report generation environment 100 areconfigured to provide various logic (e.g. code, instructions, functions,routines and/or the like) and/or services related to the generation ofinteractive reports. In some example embodiments, a natural languagegeneration system, such as interactive report generation system 102, isconfigured to generate phrases, sentences, text or the like which maytake the form of natural language text based on raw input data 110 andoutput the phrases, sentences, text or the like in the form ofinteractive reports 120 (e.g., output reports). The interactive reportgeneration system 102 comprises a document planner 130, a microplanner132 and/or a realizer 134. Other natural language generation systems maybe used in some example embodiments, such as a natural languagegeneration pipeline as described in Building Natural Language GenerationSystems by Ehud Reiter and Robert Dale, Cambridge University Press(2000), which is incorporated by reference in its entirety herein.

In some examples, interactive report generation system 102 may beconfigured to populate or otherwise instantiate one or more messagesbased on data or information in a primary data feed, the one or morerelated data feeds, the historical data, the contextual data feed, oneor more events and/or the like. In some examples, messages are languageindependent data structures that correspond to informational elements ina text and/or collect together underling data in such a way that theunderlying data can be linguistically expressed. In some examples,messages are created based on a requirements analysis as to what is tobe communicated for a particular scenario (e.g. for a particulardomain). A message typically corresponds to a fact about the underlyingdata (for example, the existence of some observed event) that could beexpressed via a simple sentence (although it may ultimately be realizedby some other linguistic means). For example, to linguistically describewind, a user may want to know a speed, a direction, a time period or thelike, but also the user wants to know changes in speed over time, warmor cold fronts, geographic areas and or the like. In some cases, usersdo not even want to know wind speed; they simply want an indication of adangerous wind condition. Thus, a message related to wind speed mayinclude fields to be populated by data related to the speed, direction,time period or the like, and may have other fields related to differenttime points, front information or the like. The mere fact that windexists may be found in the data, but to linguistically describe “lightwind” or “gusts” different data interpretation must be undertaken as isdescribed herein.

In some examples, a message is created by the interactive reportgeneration system 102 in an instance in which the data in the one ormore data feeds, may warrant the construction of such a message. Forexample, a wind message would only be constructed in an instance inwhich wind data was present in the raw input data. Alternatively oradditionally, while messages may correspond directly to observationstaken from the raw data input, others, however, may be derived from theobservations by means of a process of inference. For example, thepresence of rain may be indicative of other conditions, such as thepotential for snow at some temperatures.

In some example embodiments, one or more messages may be identified orotherwise be predefined to define or otherwise include information thatindicates which of the one or more messages are to be interactive (e.g.,a flag, an indicator bit or the like) when realized. In other words,when realized, the identified message or messages may be selected by theuser, such as via a hyperlink, to prompt the interactive responsedescribed herein. The messages may be marked based on a markingspecification (e.g., a set of rules that defined the messages that arebe hyperlinked) as defined by the domain model, based on an expertsystem or the like. In other examples, messages may be dynamicallyidentified for hyperlinking based on a determined importance level,based on one or more features in one or more received data channels,based on a learning system or the like.

The concepts and relationships that make up messages may be drawn froman ontology (e.g. a domain model) that formally represents knowledgeabout the application scenario. For example, message structures may bedefined by the domain model 114 based on a particular alert conditionand/or the raw input data, such as but not limited to the primary and/orrelated data feeds. Messages may also be derived from another datastructure, may be user defined and/or the like. Each type of message mayalso be represented by a message template, which expresses arelationship between instances of a number of concepts; the messagetemplate contains slots which may be filled in, or instantiated, usingparticular values that are derived from the raw input data.

As such, the interactive report generation system 102 is configured toinstantiate a plurality of messages based on the one or more data feeds,such as the one or more data feeds received via raw input data 110. Inorder to instantiate the one or more messages, the importance level ofeach of the messages and relationships between the messages, theinteractive report generation system 102 may be configured to access thedomain model directly or indirectly. The domain model may containinformation related to a particular domain or industry. In someexamples, the domain model may provide importance levels, single datafeed limits related to normal behaviors in a domain (e.g. normalranges), information related to anomalous behaviors and/or the like. Inother examples, the domain model may describe relationships betweenvarious events and/or phenomena in multiple data feeds. For example in aweather domain, a domain model may indicate or otherwise instantiate anextreme weather message in an instance in which wind speeds that arerelated to hurricane type events or temperatures that may cause harm tohumans or other animals or may cause damage or interference to shippingare present in the data. The extreme weather message may then be labeledas important, whereas typical temperatures or a typical wind message maynot be marked as important in some examples. Alternatively oradditionally, the domain model may be configured to contain or otherwisehave access to the diagnostic model.

In some example embodiments, the interactive report generation system102 may be configured to annotate messages with an indication of theirrelative importance; this information can be used in subsequentprocessing steps or by the interactive report generation system 102 tomake decisions about which information should be conveyed and whichinformation may be suppressed, such as by using the domain model. Insome examples, marking a message with as being related to a downstreamhyperlink may be related to the importance level. The interactive reportgeneration system 102 may assign an importance level to the one or moremessages based on the pattern itself (e.g. magnitude, duration, rate ofchange or the like), defined constraints (e.g. defined thresholds,constraints or tolerances), temporal relationships between the patternin the primary data feed and patterns in other related data feeds and/orthe like. For example, a heart rate over 170 beats per minute, or 100mile per hour winds, may be assigned a high level of importance. In someexamples, messages that describe other patterns and/or constraints maybe defined by the domain model. Alternatively or additionally, theinteractive report generation system 102 may also be configured toannotate messages with information about how they are related to eachother; for example, the interactive report generation system 102 mightindicate that an event described in one message is assumed to have beencaused by the event described in another message.

Using the importance level, the interactive report generation system 102may assign certain ones of the messages that describe or are otherwiseare instantiated with patterns or other data in the primary data feed asincluding key events. A key event may be selected or otherwiseidentified based on a pre-determined importance level threshold, such asa threshold defined by a user, a constraint defined by the domain model,or the like. Alternatively or additionally, key events may be selectedor otherwise identified based on those patterns in the primary data feedwith the highest level of importance, those patterns that exceed orotherwise satisfy the pre-determined importance level threshold and/orthe like. For example, a domain model or user preference may indicatethat any messages having wind readings over 50 miles per hour may bedesignated as key events, whereas in other examples only a message withhighest wind reading over a defined time period may be a determined toinclude a key event. In further examples, the importance leveldetermination may be performed over a plurality of time scales that maybe user defined, defined by the domain model or the like (e.g., onehour, one day, one week, one month and/or the like).

In some example embodiments, the interactive report generation system102 may also be configured to determine the importance of messages thatdescribe patterns or events detected in one or more secondary or relateddata feeds. In some examples, the interactive report generation system102 may determine one or more messages that describe patterns or eventsin the related data feeds that overlap time-wise or occur within thesame time period as the patterns in the primary data feed. For example,during the same time period as rain is detected, another data feed maydetect temperature falling below the freezing point. The interactivereport generation system 102 may then mark the one or more messages thatdescribe patterns or events in the related channels as important,expected, unexpected or as having or not having some other propertybased on the domain model. For example, the domain model may suggestthat the one or more patterns in the related data feed were expected torise as they did in the primary channel. By way of example, as winds arerising, a wave height may then be expected to rise. In other cases, thebehavior of the one or more related channels may be unexpected or may beanomalous when compared to the behavior of the primary data feed.

The one or more messages may be marked as including significant eventsbased on the importance level, domain model, constraints, user settingsor the like. For example, messages that include patterns or events inthe related data feed that have an importance level above apredetermined threshold defined by the domain model, a user or the like,and may be marked as including significant events. In some exampleembodiments, messages including unexpected patterns or messages may alsobe categorized as significant events as they are suggestive of aparticular condition or fault. Other messages including patterns orevents may be determined to be significant events based on one or moreconstraints on channel value (e.g. expected range of values or thelike), data anomalies, patterns marked as neither expected or unexpectedthat satisfy an importance level, and/or the like.

In some example embodiments, the interactive report generation system102 may also be configured to determine the importance of messages builtor otherwise instantiated using historical data, such as historicaldata, background information, event data, and/or the like. For example,historical data may contain information related to a previous conditionand the actions taken or a result.

In further example embodiments, the interactive report generation system102 may be configured to generate one or more messages based ondetermined or otherwise inferred events from the one or more data feeds,historical data, event data and/or the like. Events may include specificactivities that may influence the one or more key events and/or may havecaused the one or more significant events. In some examples, the one ormore events may be inferred based in context with the one or morepatterns in the primary and/or related data feeds. Alternatively oradditionally events may be provided as a separate channel, such as acontextual data feed or may be provided directly to the interactivereport generation system 102.

In some examples, raw input data may be received, such as the data inthe following table, that illustrates a primary data feed (e.g. heartrate) and a related data feed (e.g. respiration rate):

Time Heart Rate Respiration Rate 1 68 14 2 72 15 3 70 14 4 70 14 5 69 166 72 15 7 73 16 8 68 13 9 70 14 10 71 15 11 90 14 12 110 14 13 118 14 14116 15 15 105 15 16 92 14 17 86 13 18 80 14 19 75 14 20 72 15 21 70 1422 71 13 23 69 13 24 71 14

As is demonstrated by the raw input data in the table above, heart ratewent above 115 beats per minute (bpm) at time point 13, thus causing analert condition. An indication of an alarm condition may be received,such as by a patient monitoring system, patient monitoring equipmentand/or based on the determination that the data indicates an alertsituation. In response to the alert condition, the heart rate data feedmay be designated the primary data feed. In other embodiments, a user,the domain model or the like may indicate that the primary data feed isthe heart rate data feed. In some example embodiments, the rapid changeof heart rate between time point 10 and time point 11 lasting to timepoint 15 may be detected for use by the interactive report generationsystem 102.

A secondary or related data feed (e.g. respiration rate) may bedetermined to have a pattern (e.g. no change when a change is generallyexpected) in a corresponding time period. In some examples, thecorresponding time period may be the same time period or may be a latertime period when compared to the time period of the key events. Further,the corresponding time period may, in some examples, be defined by adomain model, such as domain model. In some example embodiments, therelatively flat and/or steady respiration rate between time point 10 andtime point 15 may be abstracted for use by the interactive reportgeneration system 102.

In some example embodiments, the interactive report generation system102 is configured to generate one or more messages based on the rawinput data in the one or more data feeds. Using the heart rate example,a message may include portions of the raw input data, to includeabstractions of the data, but may also include additional distinctionsnecessary for the generation of text as the raw input data is likely tobe insufficient for such a purpose. For example, a HeartRateSpikemessage may be instantiated using the raw input data and such a messagemay include: a time and relative variation in terms of heart rate changeor peak heart rate, a time period and a direction. In some examples,another message may be generated on related channels, historic data,events and/or the like. In some examples, the HeartRateSpike message maybe related to an Alert Message that contains information relating to thealert itself. For example, in an instance in which caffeine was appliedprior to the heart rate spike, a message may be generated to identifysuch an event. Such a message may be an Event message that isinstantiated with an event time and an event description, such as fromthe event log 116; for example, a message that indicates that caffeinehad been orally administered prior to the spike in heart rate. Othermessages such as RespirationRate (e.g. respiration rate stable=yes),HeartRateAlertHistorical (e.g. previous alert condition quantity=2,time=yesterday), HeartRateHistorical (e.g. heart rate trend=no change,time period=10 days) may be instantiated to include information aboutthe related data feeds and/or historical data. Alternatively oradditionally, the interactive report generation system 102, the documentplanner 130 and/or the like may be configured to generate the one ormore messages. In some examples, the one or more of the HeartRateSpikemessage, RespirationRate, HeartRateAlertHistorical, HeartRateHistoricalmay be marked so as to be hyperlinked in the output text.

The document planner 130 is configured to input the one or more messagesthat are generated and/or instantiated by the interactive reportgeneration system 102. The document planner 130 is further configured todetermine how to arrange those messages to describe the patterns in theone or more data feeds derived from the raw input data. The documentplanner 130 may comprise a content determination process that isconfigured to select the messages, such as based on the decisions of therecommendation and diagnosis processor.

The document planner 130 may also comprise a structuring process thatdetermines the order of messages to be included in a natural languagetext. In some example embodiments, the document planner 130 may accessone or more text schemas for the purposes of content determination anddocument structuring. A text schema is a rule set that defines the orderin which a number of messages are to be presented in a document. Forexample, an event message (e.g. medication injection) may be describedprior to a key event message (e.g. rise in heart rate). In otherexamples, a significant event message (e.g. falling respiration rate)may be described after, but in relation to, a key event message (e.g.rise in heart rate). By way of further example a document plan mayinclude, but is not limited to, an AlertMessage, a HeartRateSpikemessage and then a RespirationRate message. An Event message,HeartRateAlertHistorical message and HeartRateHistorical message maythen follow in the example document plan.

In other examples, the document planner may define one or more messagesto be interactive messages. In such an example, the document planner maycomprise logic that flags or otherwise identifies certain messages forhyperlinking if they are present in the message store based on themarking specification or the like. For example, the document planner mayflag a temperature spike message, a historic service call message or thelike for hyperlinking, if it is present in the message store and isincluded in the document plan.

In further examples, the document plan may further define the resultantaction when one or more hyperlinks are selected. In other words, thedocument plan may define (e.g., messages to be included in theinteractive response the arrangement thereof) the communicative goal ofa particular message and therefore define an interactive response. Forexample, the document plan may include a section relating to historicalactivity of a particular piece of equipment. Such a section may berealized as a single sentence in the initial report. The document planmay also contain additional detail on what would be included in asubsequent text and how a subsequent text may be arranged should theuser select the sentence on historical activity via a hyperlink.

Alternatively or additionally, the document plan may identify that agraph is to be generated in response to a selection of a temperaturespike message. The document plan may therefore define the scale, contentand the like of the graph. Alternatively or additionally, the documentplan may identify messages that were not included in the document planand provide a user with a hyperlink to information in the data that wasnot included in the document plan.

The output of the document planner 130 may be a tree-structured objector other data structure that is referred to as a document plan. In aninstance in which a tree-structured object is chosen for the documentplan, the leaf nodes of the tree may contain the messages, and theintermediate nodes of the tree structure object may be configured toindicate how the subordinate nodes are related (e.g. elaboration,consequence, contrast and/or the like) to each other. A sample documentplan may include, but is not limited to, document plan 250 of FIG. 2.Document plan 250 may include but is not limited to one or moremessages, such as message 252.

In some example embodiments, the microplanner 132 is configured tomodify a document plan, to create a text specification for input into arealizer. As is shown in some examples, a document plan may contain oneor more leaf nodes that contain messages. An example message maycomprise a plurality of slots that contain a named attribute and a value(e.g. channel and “HeartRate”). A message may also comprise slots thatcontain a named attribute and a set of named attributes and theirvalues. Other messages may include additional named attributes andvalues.

Initially and in some example embodiments, the text specification mayinclude a tree structure that matches or is otherwise structured in thesame or similar manner as a document plan tree. In some examples, one ormore messages may be combined (e.g. one or more document plan nodes) toform a single phrase specification (e.g. to form a single textspecification node). Each leaf node of a text specification may includea phrase specification with one or more empty elements. The microplanner132 may be configured to populate those element values by applying genreparameters, lexicalization rules, reference rules, aggregation rules andthe like.

In some example embodiments, the microplanner 132 may be configured toinput a series of genre parameters that are representative of genreconventions. Genre conventions are rules about the use of language whichapply throughout texts in that particular genre. In some examples,however, the rules may be overridden by a user, by lexicalization rulesand/or the like. The genre conventions specify default behavior for therealizer so that these aspects of language use do not have tocontinually re-specified by a user. Examples of genre parametersinclude, but are not limited to, the particular tense (e.g. past,present or future) that should be used consistently throughout the textto be generated; a convention on the use of pronouns in the text to begenerated; and/or a convention as to whether or not abbreviated namesare to be used in the text to be generated. Alternatively oradditionally, other elements of the phrase specification may be set bythe one or more genre conventions.

Genre conventions may be applied by the microplanner 132 as a first stepin the initialization of the phrase specification that corresponds to anindividual message. In such a case, subsequently applied lexicalizationrules may override the results of application of the genre parameters.Alternatively or additionally, genre parameters may be applied by themicroplanner 132 once all the lexicalization rules have been applied toa given message. In such a case, the genre parameters are configured topopulate the elements of the phrase specification that have not beenspecified or otherwise populated by the lexicalization rules. Forexample, a tense equal to past, may be set by genre parameter and/or alexicalization rule.

In additional example embodiments, one or more lexicalization rules maybe input. Lexicalization rules are rules that determine how the contentof individual messages may be mapped into phrase specifications. In someexamples, lexicalization rules may include, but are not limited to,message-level rules that are configured to apply to messages as a whole.Lexicalization rules may also be configured to apply to one or moreslots within each message. For example, message-level rules may specifyhow the overall form of a phrase is to be constructed from the contentsof a message (e.g. heart rate is rising, falling or staying steady).Slot-level rules may specify how specific kinds of entities that arepresent in a message should be described (e.g. heart rate is expressedvia a prepositional phrase such as “to 118 bpm”) or otherwise referredto (e.g. refer to a machine by its machine ID or full machine title).For example a message-level rule may map a name value and high ratevalue from a message to a phrase specification.

For a given domain, there may be at least one message-levellexicalization rule for each type of message in the ontology for thatdomain that may be applied b. The one or more lexicalization rules for amessage type define one or more constraints that are configured to testthe message itself, the discourse model (e.g. a model that is configuredto store the relevant aspects of the discourse context, such as a listof entities mentioned in the text so far, and the lexicalization of theprevious sentence in a text), parameters set by the document planner 130and/or the genre parameters. In an instance in which the one or morelexicalization rules matches the constraints, a default lexicalizationrule may be defined for each message type and/or slot type.

In one example, a message-level rule may be configured to specify acanned text string to be used whenever a message of the specified typeis received as input. For example, a GREETING message might result inthe simple text string “Hello friend”. Message-level lexicalizationrules may also be configured to assign the contents of the slots of amessage to particular syntactic constituents (e.g. a word or group ofwords that function as a single unit, such as a noun phrase, a verbphrase, a prepositional phrase or the like, within a hierarchicalstructure) in a sentence as represented by a phrase specification. Forexample, a lexicalization rule, or the one or more lexicalization rules,may be configured to specify the verb to be used to express a particulartype of message, and slots in the message might be assigned to thesubject and object positions in the sentence. In some examples, a usermay allocate information in the one or more slots of a message to theelements of a phrase specification by using the following non-exhaustivelist of syntactic constituents, subject: typically the first position inthe sentence; verb: the main action described in the sentence; object:typically the position following the verb; indirectobject: used in thosecases where a verb has three arguments, as in “John gave the cat abath”; frontmodifier: used to provide information that will be placed atthe beginning of the sentence, as in “yesterday, John gave the cat abath”; premodifier: used to provide information that will be placedimmediately in front of the verb, as in “John reluctantly gave the cat abath”; postmodifier: used to provide information that will be placedimmediately after the object, as in “John took a bus to the city” and/orthe like. Alternatively or additionally, a slot-level rule may beconfigured to specify a canned text string when a slot of a specifiedtype is received and/or specify a slot to be mapped to a particularsyntactic constituent in a sentence as represented by a phrasespecification.

Alternatively or additionally, a message-level rule may also specifyparticular syntactic features of the sentence to be generated, such asby overriding default values for those features either as provided bythe realizer itself or by the genre parameters. Typical features includebut are not limited to tense, which may be set to PAST, PRESENT orFUTURE; aspect, which may be set to PERFECTIVE or PROGRESSIVE; passive,which may be set to either TRUE or FALSE; negation and/or the like. Insome example embodiments, a slot-level rule may specify a particularfeature of a sentence to be generated, such as by overriding a defaultvalue. Alternatively or additionally, tense and aspect may be computed,such as by using a Reichenbachian model which is based on the time ofthe message (e.g. when the event described by the message happened), thetime the text is generated, and/or a reference time. In some examples,reference time can be computed using one or more of the followingnon-exhaustive list: setting a reference time to the time of theprevious message in the text specification, setting the reference timeas the time of the first message expressed in a current paragraph and/orthe like.

In some example embodiments, the microplanner may also apply slot-levelrules. Slot-level rules may be applied to each slot in each message toenable the slot to be mapped to an element of a phrase specification. Insome example embodiments, the message-level rules described herein mayalso be expressed as slot-level rules, allowing recursive embedding.However, in some examples the value of the slot itself may be used tofill corresponding element in a phrase specification.

In some examples, the microplanner is configured to determine whethertwo or more phrase specifications can be combined togetherlinguistically to produce a more complex sentence. For example, one ormore other phrase specifications may be combined with phrasespecification to form a more complex sentence. In some examples, areference system is configured to determine how to refer to an entity sothat it can be unambiguously identified by the reader. For example, in afirst sentence “John Smith” may be used where “he” or “his” may be usedin subsequent sentences.

Alternatively or additionally, a slot-level rule may be executed. Insuch cases, the slot-level rule may specify how the value of the slotshould be described based on the reference rules. Possible referencerules include, but are not limited to, StringValue: indicating that astring value associated with the object should be used to refer to theobject; NamedEntity: indicating that a predefined reference strategy fornamed entities should be used to refer to the object and may include thechoice between a full name or description, a reduced form ofdescription, or a pronoun, on the basis of information about the otherentities that have been referred to in the text; NumericValue:indicating that a predefined strategy for referring to numeric valuesshould be used; TimeValue: indicates that a predefined referencestrategy for referring to time values should be used to refer to theobject; DurationValue: indicating that a predefined reference strategyfor referring to durations should be used to refer to the object;EnumValue: indicating how specific values of an enumerated type shouldbe expressed and/or the like.

In some example embodiments, the microplanner may also use a slot-levelrule to specify content for each of a number of syntactic constituentswithin a linguistic element that is to be realized as a noun phrase. Forexample, the following non-exhaustive example list of positions may beavailable: determiner, specifier, noun, modifier, premodifier,postmodifier and/or the like. In some examples, a slot-level rule mayalso contain conditions that determine its applicability; amongst otherthings, these may be used to determine when the rule should have a nulloutput, resulting in the constituent being elided in the sentence beingplanned.

In some example embodiments, the microplanner may also use one or moreslot-level rules to specify syntactic features. For example, a slotlevel rule may specify the following non-exhaustive example list ofsyntactic features: a pronominal (e.g. force a use of a pronoun), number(e.g. singular or plural), an indication of definite or indefiniteand/or the like.

In some examples, after one or more messages have been marked as havinga hyperlink at the message level or in the document planner level, amicroplanner may be configured to select the particular words, phrasesor the like to be hyperlinked:

-   -   1. If, for example, there is a single message realized in a        sentence.        -   a. Hyperlink the entire sentence in some examples.    -   2. If, for example, there are two or more messages that are        realized in a sentence        -   a. For each sentence, in some examples:            -   i. Identify a subset of noun phrases, verb phrases                and/or other sentence constituents that correspond to a                particular message based on one or more predetermined                conditions; and/or            -   ii. Hyperlink the selected noun phrases, verb phrases                and/or other sentence constituents.

In other examples, words, phrases, sentence constituents or the like maybe individually hyperlinked. In some cases, the hyperlink may provide atechnical definition based on the context of the report, whereas otherexamples may include historical information for a particular entity,etc. As such, one or more lexicalization rules may be used to enablehyperlinking for the word, phrase, sentence constituent, or the like.For example, a lexicalization rule may hyperlink each entity (e.g., apiece of equipment based on a defined hierarchy). In other examples, alexicalization rule may identify particular sentence constituents basedon a listing of terms that are available to be defined, based on alisting of available additional media (e.g., additional pictures, graphsor the like), or the like.

The output of the microplanner 132, in some example embodiments, is atree-structured text specification whose leaf-nodes are phrasespecifications, and whose internal nodes express rhetorical relationsbetween the leaf nodes. A tree-structured text specification mayinclude, but is not limited to text specification 260 of FIG. 2, havingone or more phrase specifications, such as phrase specification 262. Aphrase specification may correspond to a sentence or a sub-sentencefragment (e.g. a title) and are produced from one or more messages. Aphrase specification is configured to contain one or more syntacticconstituents (e.g. subject, verb, prepositional phrase and/or the like)and one or more syntactic features (e.g. tense).

A realizer 134 is configured to traverse the tree-structured textspecification to express the tree-structured text specification innatural language. The realization process that is applied to each phrasespecification in a text specification makes use of a grammar whichspecifies the valid syntactic structures in the language and furtherprovides a way of mapping from text specifications into thecorresponding natural language sentences. The output of the process is,in some example embodiments, a well-formed natural language text. Insome examples, the natural language text may include embedded mark-up.The output of the realizer 134, in some example embodiments, is theoutput text, interactive response, interactive report or the like. Therealizer may also output situational analysis text or a narrative thatis configured to describe or otherwise summarize the one or more keyevents, the one or more significant events, the one or more contextualdata feed s, and/or the one or more events.

In some examples, once realized, the report may be shown a screen via auser interface. The user interface may provide for a hyperlink, such asbased on the word selection by the microplanner, in the form of anunderline, box or the like. The hyperlink may be selectable, in someexample embodiments, and may generate the interactive response, such asanother text graph or the like. Alternatively or additionally, thehyperlink may provide a menu of potential responses in response to aclick, thus prompting the user to determine whether a graph or text isdesired. In some examples, the interactive response is dynamicallygenerated based on the user's selection, whereas in other examples itmay be generated, but not displayed, at the same time as the initialreport.

By way of example, the realizer may output the following text inresponse to the text specification shown above (this example includingone or more hyperlinks indicated by the underlined text):

-   -   John Smith's heart rate monitor sounded an alarm at 10.56        because his heart rate went above 115 beats per minute (bpm).        His respiratory rate and oxygen saturation did not change.        Caffeine, which can affect heart rate, had been orally        administered to John at 10.54. This alarm had gone off twice        yesterday, but in both cases heart rate quickly reverted to 70        bpm. John's heart rate has not shown any long-term upward or        downward trends since he was admitted 10 days ago. John's heart        rate increase was likely caused by the administration of the        caffeine.

By way of further example, a selection of “heart rate went above 115beats per minute” may result in a graph that illustrated heart rate overtime, a text explaining heart rate over time or both. In other examples,selecting “alarm had gone off twice yesterday” may result in a text thatelaborates on the two previous alarm events. As indicated above, theelaborated text (e.g., interactive response) may be generated based on apart of the document plan or a separate document plan with a focus orcommunicative goal of describing historic events.

Alternatively or additionally, the interactive report generation system102 may be configured to generate a graph to display one or more keyevents that are detected in a data feed. In some example embodiments,the graph may also include one or more significant events in one or morerelated feeds and/or events. In further examples, a time period orduration of the data shown in the graph may be selected such that thedisplayed graph illustrates the portion of the data feed that containsthe one or more key events. The output graph is further configured toinclude textual annotations that provide a textual comment, phrase orotherwise is configured to explain, using text, the one or more keyevents, the one or more significant events and/or the events in acontextual data feed in natural language. In further examples, thetextual annotations are generated from the raw input data and furtherare designed, in some examples, to textually describe identifiedpatterns, anomalies and/or the context of the graph. In some examples, anarrative (e.g. situational analysis text) may be included with thegraph that provides situational awareness or an overview of thedata/patterns displayed on and/or off of the graph. Other outputmodalities may be included in an output in other example embodiments.

Example System Architecture

FIG. 3 is an example block diagram of an example computing device forpracticing embodiments of an example interactive report generationsystem. As described herein, the interactive report may be describedherein as an output report. An interactive response may be the graph,text or the like that is generated in response to a selection of ahyperlink and may be referred to as the another report herein.

In particular, FIG. 3 shows a computing system 300 that may be utilizedto implement an interactive report generation environment 100 having aninteractive report generation system 102 including, in some examples, adocument planner 130, a microplanner 132 and/or a realizer 134; and/oran optional user interface (not shown). One or more general purpose orspecial purpose computing systems/devices may be used to implement theinteractive report generation system 102. In addition, the computingsystem 300 may comprise one or more distinct computing systems/devicesand may span distributed locations. In some example embodiments, theinteractive report generation system 102 may be configured to operateremotely via the network 350. In some example embodiments, apre-processing module or other module that requires heavy computationalload may be configured to perform that computational load and thus maybe on a remote device or server. In some examples, interactive reportgeneration environment 100 may be offered using a software as a servicemodel. Furthermore, each block shown may represent one or more suchblocks as appropriate to a specific example embodiment. In some casesone or more of the blocks may be combined with other blocks. Also, theinteractive report generation system 102 may be implemented in software,hardware, firmware, or in some combination to achieve the capabilitiesdescribed herein.

In the example embodiment shown, computing system 300 comprises acomputer memory (“memory”) 301, a display 302, one or more processors303, input/output devices 304 (e.g., keyboard, mouse, CRT or LCDdisplay, touch screen, gesture sensing device and/or the like), othercomputer-readable media 305, and communications interface 306. Theprocessor 303 may, for example, be embodied as various means includingone or more microprocessors with accompanying digital signalprocessor(s), one or more processor(s) without an accompanying digitalsignal processor, one or more coprocessors, one or more multi-coreprocessors, one or more controllers, processing circuitry, one or morecomputers, various other processing elements including integratedcircuits such as, for example, an application-specific integratedcircuit (ASIC) or field-programmable gate array (FPGA), or somecombination thereof. Accordingly, although illustrated in FIG. 3 as asingle processor, in some embodiments the processor 303 comprises aplurality of processors. The plurality of processors may be in operativecommunication with each other and may be collectively configured toperform one or more functionalities of the interactive report generationsystem as described herein.

The interactive report generation system 102 is shown residing in memory301. The memory 301 may comprise, for example, transitory and/ornon-transitory memory, such as volatile memory, non-volatile memory, orsome combination thereof. Although illustrated in FIG. 3 as a singlememory, the memory 301 may comprise a plurality of memories. Theplurality of memories may be embodied on a single computing device ormay be distributed across a plurality of computing devices collectivelyconfigured to function as the interactive report generation system. Invarious example embodiments, the memory 301 may comprise, for example, ahard disk, random access memory, cache memory, flash memory, a compactdisc read only memory (CD-ROM), digital versatile disc read only memory(DVD-ROM), an optical disc, circuitry configured to store information,or some combination thereof.

In other embodiments, some portion of the contents, some or all of thecomponents of the interactive report generation system 102 may be storedon and/or transmitted over the other computer-readable media 305. Thecomponents of the interactive report generation system 102 preferablyexecute on one or more processors 303 and are configured to generateinteractive reports, output texts, etc. as described herein.

Alternatively or additionally, other code or programs 330 (e.g., anadministrative interface, a Web server, and the like) and potentiallyother data repositories, such as data repository 340, also reside in thememory 301, and preferably execute on one or more processors 303. Ofnote, one or more of the components in FIG. 3 may not be present in anyspecific implementation. For example, some embodiments may not provideother computer readable media 305 or a display 302.

The interactive report generation system 102 is further configured toprovide functions such as those described with reference to FIG. 1. Theinteractive report generation system 102 may interact with the network350, via the communications interface 306, with remote data sources 356,third-party content providers 354 and/or client devices 358. The network350 may be any combination of media (e.g., twisted pair, coaxial, fiberoptic, radio frequency), hardware (e.g., routers, switches, repeaters,transceivers), and protocols (e.g., TCP/IP, UDP, Ethernet, Wi-Fi, WiMAX,Bluetooth) that facilitate communication between remotely situatedhumans and/or devices. In some instance the network 350 may take theform of the internet or may be embodied by a cellular network such as anLTE based network. In this regard, the communications interface 306 maybe capable of operating with one or more air interface standards,communication protocols, modulation types, access types, and/or thelike. The client devices 358 include desktop computing systems, notebookcomputers, mobile phones, smart phones, personal digital assistants,tablets and/or the like.

In an example embodiment, components/modules of the interactive reportgeneration system 102 are implemented using standard programmingtechniques. For example, the interactive report generation system 102may be implemented as a “native” executable running on the processor303, along with one or more static or dynamic libraries. In otherembodiments, the interactive report generation system 102 may beimplemented as instructions processed by a virtual machine that executesas one of the other programs 330. In general, a range of programminglanguages known in the art may be employed for implementing such exampleembodiments, including representative implementations of variousprogramming language paradigms, including but not limited to,object-oriented (e.g., Java, C++, C#, Visual Basic.NET, Smalltalk, andthe like), functional (e.g., ML, Lisp, Scheme, and the like), procedural(e.g., C, Pascal, Ada, Modula, and the like), scripting (e.g., Perl,Ruby, Python, JavaScript, VBScript, and the like), and declarative(e.g., SQL, Prolog, and the like).

The embodiments described above may also use synchronous or asynchronousclient-server computing techniques. Also, the various components may beimplemented using more monolithic programming techniques, for example,as an executable running on a single processor computer system, oralternatively decomposed using a variety of structuring techniques,including but not limited to, multiprogramming, multithreading,client-server, or peer-to-peer, running on one or more computer systemseach having one or more processors. Some embodiments may executeconcurrently and asynchronously, and communicate using message passingtechniques. Equivalent synchronous embodiments are also supported. Also,other functions could be implemented and/or performed by eachcomponent/module, and in different orders, and by differentcomponents/modules, yet still achieve the described functions.

In addition, programming interfaces to the data stored as part of theinteractive report generation system 102, such as by using one or moreapplication programming interfaces can be made available by mechanismssuch as through application programming interfaces (API) (e.g. C, C++,C#, and Java); libraries for accessing files, databases, or other datarepositories; through scripting languages such as XML; or through Webservers, FTP servers, or other types of servers providing access tostored data. The raw input data 110, historical data 112, and/or thedomain model 114 may be implemented as one or more database systems,file systems, or any other technique for storing such information, orany combination of the above, including implementations usingdistributed computing techniques. Alternatively or additionally, the rawinput data 110, historical data 112, and/or the domain model 114 may belocal data stores but may also be configured to access data from theremote data sources/356.

Different configurations and locations of programs and data arecontemplated for use with techniques described herein. A variety ofdistributed computing techniques are appropriate for implementing thecomponents of the illustrated embodiments in a distributed mannerincluding but not limited to TCP/IP sockets, RPC, RMI, HTTP, WebServices (XML-RPC, JAX-RPC, SOAP, and the like). Other variations arepossible. Also, other functionality could be provided by eachcomponent/module, or existing functionality could be distributed amongstthe components/modules in different ways, yet still achieve thefunctions described herein.

Furthermore, in some embodiments, some or all of the components of theinteractive report generation system 102 may be implemented or providedin other manners, such as at least partially in firmware and/orhardware, including, but not limited to one or more ASICs, standardintegrated circuits, controllers executing appropriate instructions, andincluding microcontrollers and/or embedded controllers, FPGAs, complexprogrammable logic devices (“CPLDs”), and the like. Some or all of thesystem components and/or data structures may also be stored as contents(e.g., as executable or other machine-readable software instructions orstructured data) on a computer-readable medium so as to enable orconfigure the computer-readable medium and/or one or more associatedcomputing systems or devices to execute or otherwise use or provide thecontents to perform at least some of the described techniques. Some orall of the system components and data structures may also be stored asdata signals (e.g., by being encoded as part of a carrier wave orincluded as part of an analog or digital propagated signal) on a varietyof computer-readable transmission mediums, which are then transmitted,including across wireless-based and wired/cable-based mediums, and maytake a variety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). Suchcomputer program products may also take other forms in otherembodiments. Accordingly, embodiments of this disclosure may bepracticed with other computer system configurations.

Example Process Flow Diagrams

FIGS. 4-7 illustrate example flowcharts of the operations performed byan apparatus, such as computing system 300 of FIG. 3, in accordance withexample embodiments of the present invention. It will be understood thateach block of the flowcharts, and combinations of blocks in theflowcharts, may be implemented by various means, such as hardware,firmware, one or more processors, circuitry and/or other devicesassociated with execution of software including one or more computerprogram instructions. For example, one or more of the proceduresdescribed above may be embodied by computer program instructions. Inthis regard, the computer program instructions which embody theprocedures described above may be stored by a memory 301 of an apparatusemploying an embodiment of the present invention and executed by aprocessor 303 in the apparatus. As will be appreciated, any suchcomputer program instructions may be loaded onto a computer or otherprogrammable apparatus (e.g., hardware) to produce a machine, such thatthe resulting computer or other programmable apparatus provides forimplementation of the functions specified in the flowcharts' block(s).These computer program instructions may also be stored in anon-transitory computer-readable storage memory that may direct acomputer or other programmable apparatus to function in a particularmanner, such that the instructions stored in the computer-readablestorage memory produce an article of manufacture, the execution of whichimplements the function specified in the flowcharts' block(s). Thecomputer program instructions may also be loaded onto a computer orother programmable apparatus to cause a series of operations to beperformed on the computer or other programmable apparatus to produce acomputer-implemented process such that the instructions which execute onthe computer or other programmable apparatus provide operations forimplementing the functions specified in the flowcharts' block(s). Assuch, the operations of FIGS. 4-7, when executed, convert a computer orprocessing circuitry into a particular machine configured to perform anexample embodiment of the present invention. Accordingly, the operationsof FIGS. 4-7 define an algorithm for configuring a computer orprocessor, to perform an example embodiment. In some cases, a generalpurpose computer may be provided with an instance of the processor whichperforms the algorithm of FIGS. 4-7 to transform the general purposecomputer into a particular machine configured to perform an exampleembodiment.

Accordingly, blocks of the flowchart support combinations of means forperforming the specified functions and combinations of operations forperforming the specified functions. It will also be understood that oneor more blocks of the flowcharts', and combinations of blocks in theflowchart, can be implemented by special purpose hardware-based computersystems which perform the specified functions, or combinations ofspecial purpose hardware and computer instructions.

In some example embodiments, certain ones of the operations herein maybe modified or further amplified as described below. Moreover, in someembodiments additional optional operations may also be included (someexamples of which are shown in dashed lines in FIG. 4). It should beappreciated that each of the modifications, optional additions oramplifications described herein may be included with the operationsherein either alone or in combination with any others among the featuresdescribed herein.

FIG. 4 is a flow chart illustrating an example method for generating aninteractive response using an exemplary interactive report generationsystem. As is shown in operation 402, an apparatus may include means,such as the interactive report generation system 102, the processor 303,or the like, for identifying one or more messages to be hyperlinked inan output report, wherein the one or more messages are data structuresthat are configured to linguistically describe at least a portion of rawinput data. In some example embodiments one or more messages may beidentified or otherwise be predefined to define or otherwise includeinformation that indicates the one or more messages that are to beinteractive (e.g., a flag, an indicator bit or the like) when realized.Those messages that are marked as interactive will be those messagesthat are hyperlinked in the output text, in some examples. Alternativelyor additionally, the document planner may define one or more messages tobe interactive messages.

As is shown in operation 404, an apparatus may include means, such asthe interactive report generation system 102, the processor 303, or thelike, for determining one or more interactive responses based on the oneor more messages to be hyperlinked. In some examples, the messages mayindicate an interactive response. In some examples, the document planmay further define the resultant action when one or more hyperlinks areselected. In other words, the document plan may define (e.g., messagesto be included in the interactive response the arrangement thereof) thecommunicative goal of a particular message and therefore define aninteractive response. Alternatively or additionally, the interactiveresponse may be defined by a user, by a microplanner, may be defined inthe domain model or the like.

As is shown in operation 406, an apparatus may include means, such asthe interactive report generation system 102, the processor 303, or thelike, for determining one or more words in a phrase specification thatare related to the one or more messages to be hyperlinked. In someexamples, the one or more words that are related to the one or moremessages are determined by hyperlinking the entire sentence in aninstance in which a single message is to be realized into a singlesentence and/or hyperlinking at least one of a noun phrase, verb phraseor sentence constituent related to a message that is to be hyperlinkedbased on at least one of lexicalization rules, aggregation rules or areferring expression generator.

As is shown in operation 408, an apparatus may include means, such asthe interactive report generation system 102, the processor 303, or thelike, for generating the output report, wherein the one or more wordsare hyperlinked in the output report such that when selected at leastone of the one or more interactive responses is performed. In someexamples, once realized, the report may be shown a screen via a userinterface. The user interface may provide for a hyperlink in the form ofan underline, box or the like. The hyperlink may be selectable, in someexample embodiments, and may generate the interactive response, such asanother text graph or the like. Alternatively or additionally, thehyperlink may provide a menu of potential responses in response to aclick, thus prompting the user to determine whether a graph or text isdesired. In some examples, the interactive response is dynamicallygenerated based on the user's selection, whereas in other examples itmay be generated, but not displayed, at the same time as the initialreport.

As is shown in operation 410, an apparatus may include means, such asthe interactive report generation system 102, the processor 303, or thelike, for generating an interactive response using an interactive reportgeneration system in response to a selection of a hyperlink. In someexamples, the determined interactive response is a graph. As such, theapparatus may include means, such as the interactive report generationsystem 102, the processor 303, or the like, for detecting one or morepatterns in a data channel derived from raw input data; identifying oneor more patterns in another data channel also derived from the raw inputdata; generating one or more phrases describing the one or more patternsin the data channel and the one or more patterns in the another datachannel; and generating a graphical output based on the data channel,the another data channel and the one or more phrases, wherein the one ormore phrases are interactively annotated on the graphical output of thedata channel and the another data channel.

Alternatively or additionally, in an instance in which the determinedinteractive response is another report, the apparatus may include means,such as the interactive report generation system 102, the processor 303,or the like, for determining the document plan for the another reportbased on a document plan for the output report and the communicativegoal of the sentence containing the hyperlink and generating the anotheroutput report using an interactive report generation system.

In some examples, the output report or another output report, aninteractive response or the like is generated based on the currentcontext or other context of the user, reader or the like. For example,one or more messages can be marked as viewed by a user and thosemessages transformed into phrase specifications that are realized andhave been previously viewed will not be displayed in a future report. Inother examples, certain data channels over certain periods may be markedas viewed. As such, in instances in which an interactive response oranother report is dynamically generated, such an output may be given incontext and may have a reduced amount of information when compared tothe amount of information originally in a document plan for theinteractive response or another output report.

FIG. 5 is a flow chart illustrating an example method for displaying aninteractive response using an exemplary interactive report generationsystem. As is shown in operation 502, an apparatus may include means,such as the interactive report generation system 102, the processor 303,or the like, for displaying an output report having one or morehyperlinks surrounding one or more words, wherein the one or morehyperlinks provide an indication that an interactive response isavailable. As is shown in operation 504, an apparatus may include means,such as the interactive report generation system 102, the processor 303,or the like, for receiving an indication of a selection of a hyperlinkof the one or more hyperlinks. As is shown in operation 506, anapparatus may include means, such as the interactive report generationsystem 102, the processor 303, or the like, for determining, using aprocessor, a communicative goal for a sentence having the hyperlink anda current context of the reader. As is shown in operation 508, anapparatus may include means, such as the interactive report generationsystem 102, the processor 303, or the like, for displaying aninteractive response.

FIG. 6 is a flow chart illustrating an example method for generating thereport using an exemplary interactive report generation system. As isshown in operation 602, an apparatus may include means, such as theinteractive report generation system 102, the processor 303, or thelike, for instantiating one or more messages. As is shown in operation604, an apparatus may include means, such as interactive reportgeneration system 102, the processor 303, or the like, for arranging oneor more messages in a document plan in an order in which they are to belinguistically described in the output text. As is shown in operation606, an apparatus may include means, such as interactive reportgeneration system 102, the processor 303, or the like, for converting atleast one of the one or more messages into a text specification thatrepresents one or more data structures that are representative of asyntactic structure of a sentence As is shown in operation 608, anapparatus may include means, such as interactive report generationsystem 102, the processor 303, or the like, for applying a grammar tothe text specification to generate the output text.

FIG. 7 is a flow chart illustrating an example method for generatinggraphical annotations, such as in the case a graph or annotations are tobe generated as the output report or the interactive response (e.g., theanother report). As is shown in operation 702, an apparatus may includemeans, such as the interactive report generation system 102, theprocessor 303, or the like, for receiving an indication of an alarmcondition. In some example embodiments an alarm may cause the selectionof a primary data channel and a determination of a time period in whichthe alarm was generated. Alternatively or additionally other means maybe used to alert the apparatus to a primary data channel, such as, butnot limited to, a user action, a selection of a hyperlink, a detectedpattern in the raw input data or a data channel, a determined value inthe raw input data or a data channel, and/or the like.

As is shown in operation 704, an apparatus may include means, such asthe interactive report generation system 102, the processor 303, or thelike, for determining one or more key patterns in a primary datachannel. In some example embodiments the key patterns may be determinedbased on the time period of the alarm condition, however in otherexamples a larger or smaller time period may be selected.

As is shown in operation 706, an apparatus may include means, such asthe interactive report generation system 102, the processor 303, or thelike, for determining one or more significant patterns in one or morerelated data channels. In some example embodiments, the apparatus, maydetermine one or related channels based on one or more predefinedrelationships. In some examples, the predefined relationships may bedefined by the domain model 114.

As is shown in operation 708, an apparatus may include means, such asinteractive report generation system 102, the processor 303, or thelike, for determining one or more contextual channels to be included inthe graphical output. The one or more contextual channels may provideevents or other context that may be indicative of the cause of the oneor more key patterns and/or the one or more significant patterns. As isshown in operation 710, an apparatus may include means, such asinteractive report generation system 102, the processor 303, or thelike, for determining a time period to be represented by the graphicaloutput. In some example embodiments, the time period chosen for thegraph is the time period in which the one or more key patterns aredisplayed. As is shown in operation 712, an apparatus may include means,such as interactive report generation system 102, the processor 303, orthe like, for generating a natural language annotation of at least oneof the one or more key patterns or the one or more significant patterns.

As is shown in operation 714, an apparatus may include means, such asinteractive report generation system 102, the processor 303, a userinterface or the like, for generating a graphical output that isconfigured to be displayed in a user interface. In some exampleembodiments, the graph is configured to utilize the determined scale todisplay the primary data channel, one or more related channels havingsignificant events, natural language annotations, a narrative, eventsand/or the like. In some example embodiments and in an instance in whicha user clicks on a text annotation in the graph, a corresponding phrasein the situation analysis text may be highlighted and/or in an instancein which a user clicks on underlined phrase in the narrative orsituation analysis text, a corresponding annotation may be highlightedon the graph.

In some example embodiments described herein, the apparatus, a graph maybe generated (e.g., as an output report, output text, interactivereport, another report or the like) having a scale (e.g. amplitude(y-axis) and/or time scale (x-axis)) that advantageously displays one ormore data channels (e.g. a first or primary data channel, a secondary orrelated data channel and/or the like) that are derived from raw inputdata, one or more natural language text annotations and/or a narrativedescribing raw input data. As such, advantageously, a user viewing thegraph, in a user interface or using other viewing means, may be providedwith situational awareness with regard to the patterns shown on thegraph as well as the events and or patterns that may have influenced thepatterns shown on the graph.

In some examples, a first or primary data channel may be selected forinclusion in a graph based on a selection by a user, via a userinterface, may be selected based on the happening of a condition suchas, but not limited to, an alert, an alarm, an anomaly, a violation of aconstraint, a warning, a predetermined condition, selection of ahyperlink, based on an indication that the primary data channel isrelated to the communicative goal and/or the like.

In some example embodiments, a secondary or related data channel mayalso be selected. In some cases, there may be a plurality of secondaryor related data channels. The secondary or related data channel may beselected for inclusion in a graph based on the detection of anomalous,unexpected or otherwise flagged behavior in the second or relatedchannel. In some examples, the second or related channel is compared toone or more patterns in the primary data channel over a similar timeperiod. For example, a first data channel may indicate a rise in heartrate, whereas a second data channel may indicate a stable or even adecline in respiration rate. Generally respiration rate rises with heartrate, and, as such, a stable respiration rate is generally unexpected.In some examples, unexpected behavior may lead to a life threateningcondition, be indicative of a dangerous condition or the like.

Relationships between data channels may be defined as anomalous behaviorby a qualitative model such as a domain model. A domain model is arepresentation of information about the domain. For example a domainmodel may contain an ontology that specifies the kinds of objects andconcepts and the like that may exist in the domain in concrete orabstract form, properties that may be predicated of the objects andconcepts and the like, relationships that may hold between the objectsconcepts and the like, and representations of any specific knowledgethat is required to function in the domain. In some example multipledomain models may be provided for a single domain. Example domains mayinclude, but are not limited to, medical, oil and gas, industrial,weather, legal, financial and/or the like. Alternatively oradditionally, a plurality of related channels may be included, forexample pulse rate, oxygen levels, blood pressure and/or the like.

In some examples, patterns (e.g. a trend, spike, step or the like) maybe detected or otherwise identified in the primary data channel and/orin the one or more secondary data channels. Once a pattern is detectedin the primary data channel and/or the one or more secondary datachannels, an importance level or importance is assigned to each of thepatterns. In the primary data channel an importance level may be definedbased on thresholds, constraints, predefined conditions or the like. Inthe secondary data channels an importance level may also be assignedbased on thresholds, constraints, predefined conditions or the like,however an importance level may also be assigned based on therelationship between the secondary data channels and the primary datachannels and/or the relationships between the patterns detected in theprimary data channels and the patterns detected in the secondary datachannels. A pattern in the primary channel may be defined as a keypattern in an instance in which the importance level of the patternexceeds or otherwise satisfies a predefined importance level. Likewise,a significant pattern is a pattern in a secondary data channel thatexceeds or otherwise satisfies a predefined importance level. In someexamples, a pattern in the one or more secondary channels may also beclassified as a significant pattern if it represents an anomaly orotherwise unexpected behavior when compared with the primary datachannel.

In some example embodiments, a contextual channel may also be selected.A contextual channel is a data channel that provides a background orcircumstance information that may have caused or otherwise influencedthe one or more key patterns and/or the one or more significant patterns(e.g. proximate cause). For example, a contextual channel may indicatean event, such as a medical treatment that was applied at the time of orjust prior to the rise of the heartbeat and/or the fall or steady stateof the respiration rate. Alternatively or additionally, a plurality ofdata channels may also be selected for inclusion in a graph based on ananomaly or unexpected behavior.

Alternatively or additionally, one or more data channels may be selectedfor inclusion in a graph even though the one or more data channels arerepresentative of expected behavior. For example, in the medical domain,a medical professional may expect to see both heart rate and respirationrate on a graph even if both are behaving in expected ways, sinceexpected behavior may be indicative of an important result, namely aclean bill of health. As such, a selection of a hyperlink relating toheart rate, would provide a graph of both heart rate and respiration. Insome cases, based on the current context (e.g., whether the professionalhas seen a graph of respiration already, the respiration rate may beomitted.

In yet further example embodiments, events may also be generated fordisplay in the graph. An event may be described in a contextual channel,may be entered into an event log that is input with the raw input dataor may be inferred. For example, caffeine administration may be enteredas an explicit event in a patient record (e.g. in an event log), thecaffeine could be detected by a change in one or data channels whichrecord what medication is being administered through an IV line and/orthe caffeine administration may be inferred based on a spike in heartrate. In instances in which an event is identified that satisfies animportance threshold, the event may be displayed as a visual annotation.In an example in which a graph is displayed, events may be displayed asa vertical line. Alternatively or additionally events may be generatedas a horizontal line with indicators showing the multiple occurrences ofan event and/or the like. In other visualizations, events may bedisplayed via text, indicator or other visual outputs.

In some example embodiments, a scale may be selected for the graph basedon the primary data channel, the secondary data channel or the like. Thescale may be determined based on a time period or duration in which apattern that satisfies an importance threshold is identified, anomalousbehavior occurs in a related data channel and/or the like. Alternativelyor additionally the time period may be set by a user, may be a timeperiod that is significant or specifically identified on the basis ofproperties of the domain, communicative goal, current context or thelike. In some examples, if the user has already seen a graph of rate ofa time period, a selection of an event may only result in a graph of aheart rate surrounding the time of the event.

In further example embodiments, textual annotations and/or a narrativemay be included with the graph. The textual annotations and/or thenarrative may be provided by a natural language generation system, suchas interactive report generation system 102, that is configured togenerate one or more textual annotations in the form of sentences orphrases that describe the patterns in the data channels, expected orunexpected behavior, an event, a contextual channel and/or the like.Additionally, in some examples, the sentences or phrases may take theform of stand-alone text that provides situational awareness and/orsituational analysis of the graph. In some examples, situation analysistext may be configured to include pattern descriptions that contributeto narrative coherence, background information or the like. The textualannotations may be located on the graph, such as at the location wherethe anomalies and/or the patterns are represented in the graph.Alternatively or additionally, the narrative may be displayed on or nearthe graph in some examples. Whereas, in other examples, the narrativemay be contained in a separate file, may be generated before/after orotherwise separately from the generation of the graph or may be aseparate interactive response. In some examples, annotations may beshown as a preview. For example as a user is moving their cursor acrossthe graph, an interactive response may include a text box thatelaborates on a particular point on the graph. Alternatively oradditionally, the textual annotations and/or narrative may be providedvia speech or other available modalities.

Based on the one or more channels derived from the raw input data, thecontextual channel and/or the annotations, the graph may be generatedfor display. The graph is configured to display a time scale thatcontains those identified sections (e.g. key patterns and/or significantpatterns) in the one or more data channels, the textual annotations,additional available visual annotations and/or the like. In some exampleembodiments, user interaction with the narrative text may result in anannotation on the graphical output to be highlighted. Similarlyselection of an annotation may highlight narrative text related to theannotation. Alternatively or additionally, the annotations may include asymbol or other reference numeral that is indicative of or otherwiserelated to the narrative. For example, the narrative may indicate that afirst key pattern is indicated by an arrow, a circle, a box, a referencenumber or the like in the graph.

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art to which theseinventions pertain having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it is tobe understood that the inventions are not to be limited to the specificembodiments disclosed and that modifications and other embodiments areintended to be included within the scope of the appended claims.Moreover, although the foregoing descriptions and the associateddrawings describe example embodiments in the context of certain examplecombinations of elements and/or functions, it should be appreciated thatdifferent combinations of elements and/or functions may be provided byalternative embodiments without departing from the scope of the appendedclaims. In this regard, for example, different combinations of elementsand/or functions than those explicitly described above are alsocontemplated as may be set forth in some of the appended claims.Although specific terms are employed herein, they are used in a genericand descriptive sense only and not for purposes of limitation.

That which is claimed:
 1. A method for transforming raw input dataexpressed in a non-linguistic format into an output in a format that canbe expressed linguistically, the method comprising: identifying, usingprocessing circuitry, one or more messages to be hyperlinked in anoutput report, wherein the one or more messages are data structures thatare configured to linguistically describe at least a portion of the rawinput data; annotating, using the processing circuitry, the one or moremessages with an indication of an importance level of the one or moremessages; annotating, using the processing circuitry, the one or moremessages with an indication of a key event or significant event detectedin the one or more messages, the key event or significant event detectedbased at least in part on the importance level and a pre-determinedimportance level threshold; and generating, using the processingcircuitry, the output report comprising the indication of the key eventor significant event, wherein one or more words in a phrasespecification are hyperlinked in the output report such that when ahyperlink is selected at least one of one or more interactive responsesis performed.
 2. The method of claim 1, further comprising: including,using the processing circuitry, the one or more messages in the outputreport based at least in part on the importance level.
 3. The method ofclaim 1, wherein the output report comprises a graph displaying the keyevents or significant events.
 4. The method of claim 1, wherein theimportance level for each of the one or more messages is based in parton a pattern displayed in the one or more messages.
 5. The method ofclaim 1, wherein the importance level is determined over a period oftime.
 6. The method of claim 1, wherein the importance level of the oneor more messages is based in part on historical data.
 7. An apparatusfor transforming raw input data expressed in a non-linguistic formatinto an output in a format that can be expressed linguistically, theapparatus comprising processing circuitry and at least one memoryincluding computer program code, the at least one memory and thecomputer program code configured to, with the processing circuitry,cause the apparatus to: identify one or more messages to be hyperlinkedin an output report, wherein the one or more messages are datastructures that are configured to linguistically describe at least aportion of the raw input data; annotate the one or more messages with anindication of an importance level of the one or more messages; annotatethe one or more messages with an indication of a key event orsignificant event detected in the one or more messages, the key event orsignificant event detected based at least in part on the importancelevel and a pre-determined importance level threshold; and; generate theoutput report comprising the indication of the key event or significantevent, wherein one or more words in a phrase specification arehyperlinked in the output report such that when a hyperlink is selectedat least one of one or more interactive responses is performed.
 8. Theapparatus of claim 7, wherein the at least one memory including thecomputer program code is further configured to, with the processingcircuitry, cause the apparatus to: include, using the processingcircuitry, the one or more messages in the output report based at leastin part on the importance level.
 9. The apparatus of claim 7, whereinthe output report comprises a graph displaying the key events orsignificant events.
 10. The apparatus of claim 7, wherein the importancelevel for each of the one or more messages is based in part on a patterndisplayed in the one or more messages.
 11. The apparatus of claim 7,wherein the importance level is determined over a period of time. 12.The apparatus of claim 7, wherein the importance level of the one ormore messages is based in part on historical data.
 13. A computerprogram product comprising at least one computer-readable non-transitorymemory medium having program code instructions stored thereon, theprogram code instructions, when executed by an apparatus, causing theapparatus to: identify one or more messages to be hyperlinked in anoutput report, wherein the one or more messages are data structures thatare configured to linguistically describe at least a portion of the rawinput data; annotate the one or more messages with an indication of animportance level of the one or more messages; annotate the one or moremessages with an indication of a key event or significant event detectedin the one or more messages, the key event or significant event detectedbased at least in part on the importance level and a pre-determinedimportance level threshold; and generate the output report comprisingthe indication of the key event or significant event, wherein one ormore words in a phrase specification are hyperlinked in the outputreport such that when a hyperlink is selected at least one of one ormore interactive responses is performed.
 14. The computer programproduct of claim 13, wherein the program code instructions, whenexecuted by the apparatus, further cause the apparatus to: include,using the processing circuitry, the one or more messages in the outputreport based at least in part on the importance level.
 15. The computerprogram product of claim 13, wherein the output report comprises a graphdisplaying the key events or significant events.
 16. The computerprogram product of claim 13, wherein the importance level for each ofthe one or more messages is based in part on a pattern displayed in theone or more messages.
 17. The computer program product of claim 13,wherein the importance level is determined over a period of time. 18.The computer program product of claim 13, wherein the importance levelof the one or more messages is based in part on historical data.
 19. Thecomputer program product of claim 13, wherein one or more words in aphrase specification are hyperlinked in the output report such that whena hyperlink is selected at least one of one or more interactiveresponses is performed.